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INTRODUCTION 

Why would countries participate in international comparative survey research? Would it not be 
more benefsoial for a country to study educational achievement arid educational processes in 
the context of their own educational system, in stead of spending much time and energy to 
costly international comparisons? Can fair intemational comparisons indeed be made? Are 
not the well known educational statistics on fiscal and other resources, numbers of teachers, 
student participation, number of graduates, etc. sufficient for policy makers' purposes? 
The first question when reflecting on international comparative research is indeed the 'Why' 
question. Of course, nobody doubts the usefulness of national or regional comparisons (state 
by ^tate, district by district), or the relevance of the 'tradittonai' educattonal statistics. Many 
authors have answered the 'why' question of international comparative achievement research 
affimiatively (for example, Bathory, 1989; Postlethwaite, 1987; Purves, 1987). Their answers 
can be summarized in two main points. 

The most important reason for a country to participate in intemational studies of education is 
to improve their understanding of their own educational system. Since there are no absolute 
standards of educational achievement, comparative studies are essential to tell policy makers 
and educators how the level of achievement in their educational system is compared to other 
systems. In this way these studies can contribute in setting realistic standards for an 
educational system, as well as in monitoring its quality. 

Conparative studies may also be helpful in understanding the reasons for observed 
differences in performance, by exploring across nations the relations between school 
achievement and such factors as cumcula, amount of time spent on school wori<:, teacher 
training, classroom size, parental involvement and many other possible explanatory variables. 
So, the purpose of intemational comparative achievement studies is (i) to provide nattons with 
information about the quality of their educatk>n in relation to relevant reference groups, and (ii) 
to assist in understanding the reasons for observed differences between educational systems. 
These comparisons can be twofold (Postlethwaite, 1987a): (i) straight comparisons of effects 
of education, that is of total scores or subscores in international tests, which are reflecting the 
cum'cula of participating countries; and (iD comparisons of how well a countries intended 
cum'culum ('what should be taught in a particular grade') is implemented in the schools and 
achieved by pupils. 

So far, the 'why' question is discussed from the perspective of possible interest of educational 
policy makers. But also many researchers do have their arguments for conducting 
intemational comparative research. They are interested in understanding the functioning of 
education by investigating and explaining differences that exist between educational systems. 
If these educational systems are schools, theri national comparative studies across schools or 
school districts may be appropriate to find explanations for some differences. Hr ^er, if we 
are concerned v;ith national educational systems, then intemational comparison. ^ needed 
to explore the differential effects of independent variables on school achievement. The survey 
type of comparative studies, either explanatory and/or descriptive in character, takes research 
out of the laboratory and into the real world; it sees the world as a 'naturally existing 
laboratory' (Pun/es, 1989). 



In summary, there is a common interest of educational policy makers and of educational 
researchers in International comparative studies of educational achievement and in possible 
explanations for observed differences. This common interest makes this kind of studies 
valuable instruments for monitoring the educational quality, that is for supporting endeavors 
directed at improving the quality of education. 

lEA, the International Associatfon for the Evaluation of Educational Achievement, performs 
the type of research that provides data for policj' makers that can serve as a basis for 
decision making about improvements of educatfon. The remaining of this paper describes in 
short lEA's mission and history, the design and structure of a typical lEA achievement study 
some exemplary results of these studies, and lEA's future plans, organization, and relation to 
other, natfonal and international, organizalfons. 



lEA: MISSION AND HISTORY 

The International Association for the Evaluatfon of Educatfonal Achievement (lEA) is an 
independent international cooperative of research centers from about 45 countries. It has 
taken as fts mission the conduct of comparative studies focusing on educatfonal policies and 
practices in order to enhance learning within and across sytems of education. lEA has 
committed ftself to a cycle of studies of learning in the base school subjects and to additional 
studies of particular interest to its members. 

In its studies iEA focuses not only on measuring educatfonal achievenrent; but it studies also 
the effects on educatfonal outcomes of groups of variables such as the cumculum, and the 
organizaUon of schools and classrooms; the relatfonsh^) between achievenrent and attitudes- 
the effects of certain subject matter practices, such as laboratory work in science, and time 
spent on in-class compositfon teaching in mother tongue; and different educatfonal practfces 
and attainment of specal groups (Purves, 1987). 

Fbunded in 1959, lEA's first intematfonal study of educatfonal achievement (mathematfcs in 
elementary and secondary educatfon, in 13 countries) was completed in 1966. IEA was the 
first international organizatfon that used thft same objectivft cognitive tfists in more than Qn«» 
QsmSL- Other studies were the Six-Subject Survey (in science, reading, literature, dvfc 
education, and English and French as foreign languages), with a varying number of countries 
fromi 8 (French) to 19 (science), completed in 1973-1974; the Second Intematfonal 
Mathematics Study (20 countries, data collectfon in 1931); the Classroom Environment Study 
(10 countnes, 1982); the Written Compositfon Study (13 countries, 198^, 1985); and the 
Second Intematfonal Science Study (25 countries, 1983); see appendix for the recent 
published international volumes on some of these studies. On-going studies are the Pre- 
Primary project, the Computers i i Educatfon study, and Reading Literacy study; while a Third 
Intematfonal Mathenfiatics study is under preparatfon. 

rtie IEA guidebook (Pelgrum, 1989) descnTjes the activities, institutfons, and people involved 
in !cA. 

A special issue of Comparative Educatfon Review (Postlethwaite, 1987b) focuses on the first 
25 years of IEA. 



DESIGN OF A TYPICAL IEA ACHIEVEMENT STUDY 

A typical IEA study starts with developing a conceptual framework that clarifies the issues to 
be addressed, suggests appropriate methods of investigatfon, results in validated measures 
of educational outcomes and processes, and uses those analytic tools that can best elucidate 
key factors ind issues (from lEA's missfon statement). We will illustrate for the populatfon of 
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13-years-olds of the Second International Mathematics Study (SIMS) some aspects of the 
design of a typical achievement study, by discussing the conceptual framework of such a 
study; the definition of populations and samples; the instruments typically used, especially the 
instrument "opportunity to learn-; and the cuniculum analysis which is underlying instrument 
development. 

Conceptual frar^iework 

!EA studies are not restricted to just surveying achievement of students. They are also 
designed to contribute to the explanation of differences in outcomes, and in this, they 
contribute to nronitoring the quality of education. The latter purpose requires that the variables 
included in a study must be derived from a conceptual model of how educational systems 
work. Shavelson, McDonnell, Oakes and Carey (1987) conclude that cun-ent research about 
schooling is not sufficently advanced to support a strictly predictive or causal model, but that it 
can provide a framework that identifies the logical relalfonships among those features. They 
provide a general model of the educational system and the variables (called 'indicators' by 
them) for measuring each component. The model consists of inputs, processes and outputs 
(see Rgure 1). Based on their research review they suggest how various elements of the 

FIGURE 1 ABOUT HERE 

system are likely to be togically or empirically related. 

An example of such a framewori< is derived from lEA's Second Intematfonal Mathematics 
Study (SIMS)as presented in Travers and Westbury (1989); see Figure 2. It distinguishes 
three types of the (mathematics) curriculum: the intended, implemented and attained 
curriculum. 

FIGURE 2 ABOUT HERE 

At the macro level of the educatfonal system (nation, region, school district), there is the set of 
intenttons for the cunriculum; the official goals, the kJeas and tradilfons of the mathematicians 
and educators. This collections of intended outcomes, together with course outlines, official 
syllabi, and textbooks, forms the intended curriculum . 

On the ievel of the school and the classroom we have the implemented curriculum , as is the 
cunrfculum 'translated into reality by the teacher* (p that is the curriculum as being taught 
by the teachers. Rnally, we have the attained cur,.oUium. the student behavior, that is the 
knowledge, skills and attitudes acquired by the students. 

An important question for monitoring the quality of education is how far the implemented 
curriculum matches with the intended cuniculum: do teachers teach the syllabus; is their 
perceptton of the intended curriculum in agreement with the intenttons at the system level; do 
they teach all intended toptos, or are they leaving out some toptos because of, for example, 
pressure of time? Similariy important is the questfon whether the students learn what they are 
expected to learn according to the intentions; and, if this is not the case, whether the 
explariafion can be found in discrepancies between the different kinds of curriculum. 
The different types of curricula are refem^ng to different curricular contexts (see Rgure 2). 
Furthermore, we have the background variables, or antecedents of the cumculum, that 
influence the curricular contexts, as well as the cum'cular contents. For example, at the 
system level, the wealth of a society may affect the retentivity of the school system; 
communrfy and home background of students may influence the amount of mathematfcs 
taught and learned. 

The model for the Second Mathematfcs Study, summarized in Figure 2, provkJed the starting 
point for the further design of the study. The rows indicate the level at which relevant data 
v/ere collected, and hence indicate the nature of variatton i.. the data. For example (Travers & 
Westbury , p.8): 



Level 



Source of variatfon 



1. System 

2. School/classroom 



Between systems 
Between schools 



3. Student 



Between classrooms in schools 
Within classrooms 



Populations and samples 

Different studies may be directed at different populations. To relate classroom and teacher 
vanab,es to achievement measures, in a typical lEA study the population definition is given in 
temre of age and grade. For example, in SIIVIS the population of lower secondary education 
was defined as: all students in the grade in which the modal numtter of students has attained 
the age of 13.0-13.11 years by the middle of the school year. 

Such a population definition allows each participating system to determine a grade level 
within which classrooms, and through it teachers and students, can be selected. lEA 
achievement studies usually have three populations, respectively at the elementary, lower 
secondary, and upper secondary education level. 

Car£iful sampling procedures are applied, striving at the same confidence intervals across 
participating countries. Each study has a sampling referee, a specialist who detemiines the 
actual sampling procedures, who has to approve the sampling plans submitted by the 
countnes, and has to create acceptable solutions for practical problems. Such procedures 
give optimal guarantees that the data collected in participating countries are indeed providing 
reliable estimates of the variables on which data will be collected. 



Instruments, Opportunity to Learn (OTL) 

In each sti'dy a variety of instruments is applied. For example, 'n SIMS on the level of 
schools, classrooms, and students: school organization questionnaire; teacher questionnaires 
on bacl(ground, attitudes, and teaching practices; students questionnaires on background and 
attitudes; and achievement tests. The achievement tests consist predominantly of multic a 
choice items. In some cases lEA experimented with other item fomiats, such as open ended 
mathematics items, essays in mother language, perfomiance tests for science, and oral 
testing of foreign language. 

Besides, teacher ratings were collected on whether the content needed to respond to each 
Item of the achievement tests had been taught that year, in prior years, or not at all, to their 
students. This measure is called Oooortunitv To Lfiam (OTL). It provides us with a second 
measure for student outcomes, a score, adjusted for OTL, next to the raw scores on the 
achievement tests. Whether to choose in reporting results the raw achievement scores, or the 
adjusted ones, depends on the purpose. If policy makers are interested in comparing their 
OTuntry with come other countries, they may be interested in the raw test scores. Generally, 
lEA IS reserved in using just raw scores, because it easily makes a study to a kind of 
Olympics' or a 'horse race'; while scores adjusted for OTL may prcvWe more insight in the 
quality of the educational processes. OTL offers in principle an important tool for carrying out 
nuanced and relevant comparisons. 

This can be illustrated with the following example of the subject French as a foreign language 
(from Postlethwaite, 1987a).ln any foreign-language testing it is usual to test the four skills of 
reading comprehension, listening comprehension, writing, and speaking, it is also common 
that these four skills are emphasized differently by teachers according either the intended 
cumculum in a country, or to how well teachers have mastered each skill. Speaking, for 
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example, fs a skill that has the greatest emphasis in some countries and the lowest in others. 
Therefore, it would be fair to compare the speaking achievement raw scores in Country A 
(with high emphasis-OTL) with the raw scores in Country B (with very low emphasis-OTL)? Or 
should the emphasis-OTL be used to create an achievement score reflecting how weil 
students perform given their opportunity to learn? (p.154). 

Development of achievement tests 

An important aspect in making valid international comparisons of achievement is the 
appropriateness or fairness of the achievement tests for the students in all participating 
systems. This implies that the test devetopment at least need'i to reflect the intended cunicula 
in these countries. The procedure for the development ot achievement tests, which lEA 
foltows to obtain optimal fairness, will be illustrated with the SIMS (see Travers & Westbury, 
1989, chapter 2). 

Starting point is that tests reflect as good '^s possible the intended cunicula of the 
partic?)ating educational systems. At the other hand, tests need to have sufficient 
commonalities of mathematica! content that meaningful comparisons between the 
partfcipating systems is possible. The organizing framework for this was a content-by- 
behavior grid. For example, the coWent dimensfon for population A (13 year oWs) consisted of 
133 entries under five broad categories: Arithmetfc, Algebra, Geomelry, Statistfcs, and 
Measurement. The behavior dimensfon was divided into four levels: computation, 
comprehensfon, applicatfon, and analysis. Each participating system was asked to describe 
their cunicula using the format of this grid. For each cell in the grid, natfonal centers were 
asked to report on: 

1. its uriivergality (i.e., whether the nn.athematfoal skills and knowledge defined by the cell 
was part of the curriculum for "all", "some", or "none" of the students in the system); 

2. the emphasis given this aspect of the cuniculum; 

3. the importance of this aspect of the curriculum within each systems cuniculum (p.1 8). 
National centers were also requested to supply items whfci". they considered appropriate for 
ten preselected cells in each grid. 

Based on the completed returns of 14 educational systems, the International Mathematics 
Committee of the study discussed the differences among the ratings and reached a 
consensus. An example of such a rating is the grid for Algebra in Table 1 . 

TABLE 1 ABOUT HERE 

Given the differences in .ratings between systems, it is clear that the cun-iculum of a particular 
systems will be reflected in the international grid more or less well, depending on the extent to 
which systems have unique emphases on certain aspects of mathematics. As the grid sen/ed 
as the basis for test construction, it was important to get some information of the "goodness of 
fit" of the grid to each system's curriculum. This informatfon was provided by comparing the 
intematfonal grid with each systems cuniculum as descn'bed in their cun-iculum rating. For 
this, each importance rating was given a numerical value: 

V (very important): 3 
I (important): 2 

Is (important in some systems): 1 
Not important or not taught: 0 

On the basis of these values, for each educational system the difference between their 
importance rating and the international rating was calculated for each topic area (i.e., row in 
fig. 2). Figure 3 presents the results for Algebra. The cross-hatched regtons indicate that the 

FIGURE 3 ABOUT HERE 
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international rating for the given cell exceeded the rating provided by the National 
Mathematics Committee for the indicated system. The black regions indicate that the rating of 
the National Committee for that topic exceeded the rating in the international grid. The width 
of the shaded region, ranging from one to three units, reflects the magnitude of the difference 
In ratings. Blank cells Indicate agreement between national and international Importance 
ratings for the corresponding topics. We see that for Algebra there is some indication that the 
international grid does rwt give enough importance to some topfcs in some systems; countries 
having notable black bars Included Finland, France, Japan, the Netheriands, New Zealand 
and Canada-Ontario. The result of the test development based on this grid was for the 
population of i3-years-olds a pool of 199 Items. 



EXAMPLARY OUTCOMES OF lEA STUDIES 

Results of several lEA studies were reported recently (see appendix). Many other publications 
are written on national and intemattonal results of these and previous studies; see for this the 
lEA bibliography (Degenhart, in press). We will restrict ourselves here to some illustrative 
examples, predominantly chosen from the SIMS results of the population of 13-years-oWs. 

Comparisons of total test scores 

Until recently lEA has always been hesitant to publish the raw total test scores. The 
publication of this type of comparison does not do justice to the careful preparation of the 
studies, and takes not into account differences between curricula of countries and in 
opportunity to learn between students. Furthermore, such "Olympics" or "horse races" cleariy 
do not serve research purposes. It Is for this reason that, for example, Robitaille and Garden 
(1989), when reporting the outcomes of school mathematics do not give these raw scores. 
Clearly, lEA is not interested in newspaper headlines such as "Sweden = Nigeria in Math" or 
"USA Bottom" (Postlethwaite, 1987a). 

At the other hand, polfcy makers are interested in some sort of yardstick, preferably a total 
score measure. This is cleariy illustrated by President Bush of the USA, who In his last State 
of the Unfon announced as one of the six national goals for education, that "by the year 2000, 
U.S. students will be first in the worid In science and mathematics achievement." The 
background document of this section of the State of the Unton states explair.s this goal as 
foltaws: "while no international comparisons of student achievement to date are consWered 
adequate, available measures suggest that U.S. 13-years-okls perfomi near the bottom in 
science and mathematics compared to their peers In other industrialized countries." (White 
House, 1990). The document further states the need for a pemnanent international framewori< 
for coordinating international assessnrents that compare the perfomiance of U.S. students in 
mathematics and science to that of their counterparts in other industrialized countries. In this 
context reference is made to the lEA Third International Mathematics Study, which is under 
preparation, and will have data collection in 1993/1994. 

So, if lEA does not publish the total scores on the achievement tests, others will do this as 
soon as the data tapes of a study have been released. For example, the total cognitive score 
of the 13-years-olds in SIMS are published by Suter (1989), in conjunction with Opportunity to 
Learn (see Table 2). It is this kind of table that is referred to by the President of the USA. 

TABLE 2 ABOUT HERE 

The OTL scores in Table 2, which are teacher judgements, show great variation between 
countries, indicating that the overiap between the test and the implemented cumculum in the 
classrooms (see Figure 2) of the respective systems varies a tot across systems. In other 
words, the OTL data suggests that the SIMS test does not perfectly reflect the implemented 
mathematics curriculum in every participating country, unless the careful preparation of the 
tests based on the intended curriculum of the participating systems. These low OTL teacher 
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scores suggest also that within many countries a substantial difference exists between the 
intended curriculum Cwhat is expected to be taught*) and the implemented cuniculum ("what 
actually is be taught'). 

Tihe differences in curriculum coverage (OTL) are strongly related to the student achievement 
score; Suter (1989) found a (Pearson product moment) conrelation of .52. If we take the 
cognitive test as a standard to which student achievement might be measured (which is an 
acceptable assumption, given the careful procedure of 'cuniculum diiven' test development), 
a possible interpretation of this strong associations, that in some countries, such as the USA, 
"students might not leam as much as in other countries because they are not presented with 
the information in class" (Suter, p.14). This was, in fact, the conclusion of the international 
mathematics committee (McKnight et a!., 1987), which explains the relatively low performance 
of U.S. sludents on mathematics achievement tests in SIMS to a cumculum that moves very 
slowly, containing huge amounts of redundancy, and introduces, compared with other 
systems, fewer new topics. 

Another example of how national achievement data can be interpreted in an international 
context is presented by Marklund (1989). When the national SIMS data were published in 
1983, little attention was paid to them. The between-school variance was very low, which 
could be interpreted as showing that the standard of mathematics education was not much 
different across Swedisch schools. But when the international averages were published in 
1985, showing poor results of Sweden (see Table 2), the SIMS became highly relevant in the 
public discussions about, for example, the refomn of teacher education. A governmental 
committee on mathematics education proposed, anrwngst others, a study day in alt schools 
compulsory for all mathematics teachers in grades 7-9; eamnarked resources during four 
yoars for inservice teacher training, and for development viork and curriculum changes. The 
low between-school variance was now politically and practically helpful. In that the measures 
proposed by the mathematics committee resulted in a national development program for all 
schools (Maridund, p.41). 

Based on the Swedish analysis of the SIMS and also of the SISS results, Mari<lund (1989) 
concludes, that it is imperative that national and international results are published and 
discussed simultaneously. If not, the intemational results will remain an Olympic game with 
very limited possiblities of being explained by and added to the understanding of national 
results (Marklund, p.42). Given the eariier discussion of some USA results, we can on^ but 
fully agree with this conclusion: total test scores, in conjunction with measures of cuniculum 
coverage like Oil, provide us with a rich source of possibilities to leam about our national 
curricula. 

Relation between achievement and other variables 

In the preceding sectfon we have seen that there is a relation between student achievement 
and teacher ratings of QIL for mathemattos. Similar results are found in other lEA studies; for 
example, by Comber and Keeves (1973) in the first science study. Pelgrum (1990) recently 
showed that the validity of these teacher ratings are in general promising, whtoh makes this 
measure a powerful instrnment for analysing student achievement in the context of the 
implennented curriculum ("what is actually taught in the classrooms'). Kifer (1989) concludes 
from analyses of the different findings on the relattonship of achievement with other variables, 
that what is taught (thus: OTL) is related to achievement differences, but im students are 
taught is not (p.57). 

Artipuntof tim? is another variable whteh is related to achievement. Carroll (1975) shows from 
the lEA study of French as a second language, ihat the tonger students study a subject, the 
more proficient they become in Jt. This amount of time coukJ be reduced if the student would 
have a teacher fluent in French, being taught mainly in French, and have serious aspiratfons 
to leam French. 

Kifer (1989) points to different implfeations of these two findings. -Where Carroll's findings 
include implication for the allocation of time, teacher competence, and teaching methodology; 



the OTL findings suggest that the relative importance of the variables places expooure to 
content very high on a list, and teaching variables and 'sequencing' of the curriculum rather 
low. ... Tills means that exposure to the material is the necessary condition for learning- while 
sequencing, pacing, and teaching variables follow in importance" (p,58/9). 

Schmidt and Klfer (1989) also concluded that in SIMS for the population of 13-years-olds 
CMS£^ has no effect on achievement (which, of course, doss not mean that there are no 
other arguments for reducing class size in many systems) 

Looking at flfil^ the results of SIMS (13-years-olds) tend to confirm those of earlier 

cll,Sfi » K ^IV^ *° P®'^®"" °" mathematics than giris (Robitaille, 1989; 
Schmidt & Krfer, 1989). A more interesting outcome is that there are systems in which giris 
are outperforming boys on all mathematics subtests (Belgium-Flemish, Belgium-French. 
Finland, Sweden, and Thailand); while in other countries the reverse is found (for example, in 
.t"^ v^'-S.^"**' ""^^ Netheriands, New Zealand, Scotland, and some 

others). The interesting lesson from such a finding Is, that a pattern what may be considered 
as given or not changeable, may seem a less unchangeable situation than many of us 
presuppose. A closer analysis of countries where giris outperform boys may lead to 
irteresting suggestions of how to change mathematics practice in countries with the reverse 
situation. 



A national system analyzed: th« case of Hong Kong 

The examplary findings of lEA studies, presented so far, are in fact all examples of results 
presented in the context of the input-process-output model (Figure i). Main effects are 
discussed (based on raw scores, and on scores adjusted for OTL). as well as relations of 
achievement with a few process and input variables. 

lEA studies may give reason for other types of analysis wHhin an educational system, in 
which one tries to understand and interpret complex relations between variables within the 
own system with the help, and in the context of international comparisons. An example 
provide Brimer and Griffin (1985) in their analysis of the SIMS data of Hong Kong (as reported 
cLc / ' A!®^- ""^^y ^^"^ particular interest to investigate why Hong Kong students in 
SIMS (population of 13-years-olds) achieved relatively high in the international comparison, 
while they were a year younger, on average, and given that measures on most important 
home background and school-related variables for Hong Kong pointed to the likelihood of tow 
average achievement. Brimer found that various factors pointed to a systematic selective 
effect having to do with the process of grade repetitton and promoting by whteh Hong Konq 
s tuderts reach seondary school. At each grade level in the primary school, about 6 percent of 
students are retained to aitow them to improve their tow achievement. This results in a wide 
age dispersion by the time they reach seondary school, and a strong negative correlation 
between age and achievement. More important, at the transitton stage from primary to 
secondary school, internal school assessments that are scaled on the basis of aptitude 
measures taken toward the end of primary schooling are used in detennlning whteh students 
Shan have the first chotoe of secondary schools. Brimer*s evidence suggested that the 
mathematfcs component of the aptitude tests played a dominant role in selectfon for 
secondary scfiool and thus "that a bias exists in the evaluatton of the mathem? :ics component 
of the cumculum, even though there is no evident bias in the time of school resources 
devoted to it. ... The combined evidence lends credibility to the view that intematfonal 
comparisons are capable of highlighting systematte disorders in an educatfon system that 
might otherwise have gone undetected" (from Garden, 1987; p.67). 
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lEA'S CYCLE OF STUDIES 



From the examplary results of lEA studies we conclude that these studies can piay an 
Important roie in nwnKoring the quality of education. The nxxJIej of the educationai system in 
Figure 1 iiiustrates the kind of data which are needed for nwnKoring activities. 
Countries can often oasiiy generate certain national educational statistics or indicators, such 
as numbers of students and teachers at each level of education, percentage of repeaters, 
numbers of graduates of the different school types, public expenditure on education, etc. 
These kind of data are intemattonaily published by, for example, Unesco. 
But Unesco and OECD are not collecting or publishing data on student achievement, on the 
curriculum and cunicular change, on classroom processes, and on student background. lEA 
type of studies are needed for generating these kind of indicators and to allow nations and 
International organizations to analyze relationships between these kind of variables, and of 
these variables and the more 'easy to collect' educational statistics. 
To meet this need, lEA has decided to plan a series of phased repeat surveys in the 
participating systems, with constant updating of the items and tests in the light of curricular 
changes, but at the same time with the possibility to study developments over time (Purves, 
1987). These surveys are focused on basic school subjects in elementary and secondary 
education : reading (data collection in the academic year 1990-91), mathematics (1993-94), 
science (1996-97), and foreign languages and cultural studies (1999-2000). Next to the cycle, 
the possibility will remain for additional studies, like the on-going Pre-Primary study, and the 
study on Computers in Education. 

The sitn^eys in basic school subjects will have an increased emphasis on opportunity to learn 
and on classroom processes; which implies that lEA will continue to take as the inportant 
starting point for their studies that the intended and the implemented curriculum of the 
participating systems has to provide the contexts for interpreting student achievement (i.e. the 
attained curriculum). Next to this, increased attention will be pakl to ways of combining 
quantitative and 'narrative' methodologies in order to provkJe potential for rich interpretations 
of the statistical data, and in this way providing decisfon makers which more comprehensive 
Information. This 'enrichment' will already be applied in the on-going studies of Reading 
Literacy and Computers in Education. In the latter, as an intemational option, per country a 
case study will be conducted in two schools from the survey sample which are known as 
belonging to the best examples of computer using schools in their cc-jntry, to get an 
understanding of the factors on school level which inJIuenced positively t!ie integration of 
computers In the educattonal processes. 

Such a cycle of studies has several advantages. Members can plan ahead for participation, 
and can budget the national studies well In advance. They also can relate natfonal 
assessment projects to lEA studies, for example by adding to the 'national' lEA studies items 
from national assessments. 

A more substantive advantage is that in a cycle, although the subject matter focus may differ 
certain variables or indicators can be studied tongitudinally across the studies, such as school 
variables, student and teacher background variables, and classroom processes. In this way 
the lEA cycle enrtohes the possiblities of studying over time the effects of interventions in 
educational systems. 



RELATIONS WITH OTHER INTERNATIONAL ORGANIZATIONS 

Nowadays, we observe not only an Increasing interest in the quality of education on national 
levels (see. for example, our discussfon of Sweden and the USA), but this interest Is also 
manifest and increasing on Intematfonal level. Examples are Unesco, with its regular 
publication of educational statistfcs of member states, and the OECD Intemational Indfcator 
Project, which attracts a tot of interest and involvement of all member countries. Furthennore, 
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In Europe the process M un Jication within the European Community Is accelerating, which as 
well calls for increa?ad Interest in indicators of educational quality in the member count.ies. 
lEA has decided to take Into account this interest of international organizations In the kind of 
Indicators generated by Its studies. Already In the Reading Literacy study, in a more informal 
way. some variables proposed by the OECD indfcators study wfli be included in the 
Instruments. From now on, intematfonal organizatfons will be invited to participate In the 
preparatfon stage of each new study; for example in invitattonal workshops and meetings of 
nattonal research coordinators where the design of a study will be elaborated, and the 
variables to be Included are detemiined. Similarly, lEA Is prepared to have these 
organizations in some v/ay or another involved in the discussion of the results of the studies. 
To be concrete, lEA m already invite representatives of OECD, Unesco, EC, and World 
Bank (and possibly other relevant instituttons) for meetings during the fall of 1990, where the 
design of the Third International Mathematfcs Study (data collectfon 1993-94), and stage il of 
the Computer in Educatton study (data collectton 1992) will discussed. 



FUTURE PLANS AND ORGANIZATION 

The cycle of studies, and the involvement ir tematfonal organizations as potential 'consumers' 
of lEA results in certain stages of the studies, are both the result of a process of reflectfon 
within lEA on how to functfon meaningfully in the future, while maintaining itf> mission. Another 
resun of lEA's reflectfon on its future will be the re-establishing of the training activities. In the 
1990 General Assembly plans will be discussed for a set of training seminars for beginning 
and advanced survey researchers, to be conducted in English, French, and Spanish. These 
plans are inspired not only by the need of lEA of having well qualified researchers as national 
research coordinators in its studies, but more generally also by the growing interest in 
monitoring the qualrty of their educatfon, both in more devetoped countries as well as in less 
developed countries. 

The announcement of these plans and the typical character of lEA studies, being 
collaborative efforts in the design of the study, the ana/ysis of the curricula, and the 
development of instruments, sets on lEA the task of carefully monitoring Its studies and other 
activities. For this purpose, a small executive office is established in the Hague, whtah is 
operatfonal from March 1990. The address is: 

lEA 

c/oSVO 

Sweellnckpleln 14 

2i>l7 GKTHE HAGUE 

the Netherlands 

tel: *3170-346967^ (also fax) 

e-mall (EARN/BITNET): SURF445 @ KUB.NL (from 1 September 1990: lEAHQ @ KUB.NL). 
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Research volumes on recent studies: 



Siim- Oxford (UK): Pergamon Press. 
Gorman, T.P Purves, A.C. and Degenhart, R.E. (Eds) (1988). The lEA study of wrtHan 
CpmpQgitfon I: the Intemattonal writing tasks anri ^ rfng scales. Oxford (UK): Pergamon 

Robltailie, D.F. and Garden, HA. (Eds) (1989). The lEA stijdv of math»mpiirs !i: contexts and 

outcomes of school mattiftrngthp^ Oxford (UK): Pergamon Press 

mi'thli ^^^^"V'}- (Jds) (1989). The lEA sfriv of mathematics I- an.yc.c 

mathematics n^im<;^ii(^ Oxford (UK): Pergamon Press, 
aher volumes on the Second Intematicnal Mathematics Study (vol. Ill), the Written 

Composition Study (vol. II and III), and the Second International Science Study wiD 

published later on in 1990 or in 1991. 
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Figure 1 : A model of the educational system (from Shavelson et al., iT7), 



Inputs 

Fiscal and 
other 
resources 

Teacher 
quaiity 

Student 
background 




Processes 

School 
quality 

Curriculum 
quality 

Teaching 
quality 

Instructional 
quality 




Outputs 

• Achievement^ 

• Participation 

• Attitudes and 
aspirations 



15 



Rgure 2: A model for the Second International Mathematics Study, SIMS (from Travers 
& Westbury) 
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Tabfel: 



Importance for instrument construction of content topics and behavioral 
categones for Algebra (population 13-years-oIds) (from Travers & VVestbuiy) 



Content topics 



Behavioral Categories' 
Computation 

Comprehension 

Apph'carion 

Anaivsis 



I 
I 

I 

Is 



Algebra 

*10l Integers v V I 

102 Rationals i { { 

103 Integer exponents . _ 

104 Formulas and algebraic expressions I I I 

105 Polynomials and rationcl expressions I Is 

106 Equations and inequations (linear only) V I I 

107 Relations and funaions * l i | - 

108 Systems of linear equations - - - - 

109 Finite s>'stems _ _ _ _ 

110 Finite sets I i i . 

1 1 1 Flowcharts and programming « - - _ 

1 12 Real numbers _ _ _ _ 

•Rating scale: V = very important; I = important; Is = important for some s>'Stenis. A dash 
(-) = not important 
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Figure 3: The match between the international grid and the national grids for Algebra 
{population 13-years-otds) (from Travers & Westbury, 1989) 
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Table 2: 



Student Cognitive Achievement and Teacher Opportunity to Leam (OTL) for 
Eighth Grade Level Mathematics Students in Fitteen Countries: Rank Order bv 
Total Cognitive Score (from Suter. 1 989) 



(Average percent of items answered correctly) 

All 157 items 30 Algebra items 

Country . 

Cognitive OTL Learning Cognitive OTL Learning 
efficiency 

M^ , X <2/3) (5/6) 

_J_ll ill (3) (4) (5) (6) (7) 

1 ^ ■'^•■^ SI 60.3 83.4 72 

Netherlands 57.1 70. 5 81 51.3 73.1 70 

Hungary 56. 0 90.1 62 50.4 90.9 55 

Belgium (Flem) 53.2 60.6 88 52.9 70.9' 75 

Sn^SJ n X ^^'^ "^2.3 73 55.0 86.1 64 

Canada (B.C.) 51.6 72.0 72 47.9 84.6 57 

Belgium (Fr) 51.3 - 49 ^ _ 

Canada (Ont.) 49. 0 71.7 68 4210 71.0 59 

Scotland 48.4 - 42 9 _ 

England § Wales 47.3 68.4 69 40.*1 63.1 64 

Finland 46.8 61.0 77 43.6 68.5 64 

nnVrir? 45.5 64.1 71 39.4 62.7 63 

S^^f States 45.3 68.6 66 42.1 68.9 61 

^Soiin ^^-2 ■'^•^ 56 37.7 83.0 45 

Sweden 41. 8 53.8 78 32.3 49.8 65 

15 Country ave. 5O.O 69.6 72 45.8 73.5 62 



^= opportunity to Learn, as 

reported by the teacher 
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